Topic Trend Detection in Text Collections using Latent Dirichlet Allocation

Algorithms that enable the process of automatically mining distinct topics in document collections have become increasingly important due to their applications in many fields and the extensive growth of the number of documents in many domains. Traditionally, the task of topic discovery has been mainly addressed through algorithms that work on a snapshot view of the repository, which ignores the temporal characteristics of the collection. In a significant number of collections, the documents are temporal in nature and this temporal dimension can influence the topic discovery process. This paper proposes a generative model based on latent Dirichlet allocation that integrates the temporal ordering of the documents into the generative process in an iterative fashion. The document collection is divided into time segments where the discovered topics in each segment is propagated to influence the topic discovery in the subsequent time segments. We conduct experiments on the collection of academic papers from CiteSeer repository. In addition to the textual content of the documents, we augment the text corpus with the addition of user queries and tags and integrate the citation graph to boost the weight of the topiPreprint submitted to Information Systems 24 December 2007 cal terms. The experiment results show that segmented topic model can effectively detect distinct topics and their evolution over time.